Skip to content

fix: fail Deployment tracking fast on ReplicaSet pod-create errors - #399

Open
Sharvash wants to merge 2 commits into
werf:1from
Sharvash:fix/398-replicaset-quota-events
Open

fix: fail Deployment tracking fast on ReplicaSet pod-create errors#399
Sharvash wants to merge 2 commits into
werf:1from
Sharvash:fix/398-replicaset-quota-events

Conversation

@Sharvash

@Sharvash Sharvash commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

This PR makes the Deployment tracker fail fast on ReplicaSet pod-create errors such as FailedCreate ... exceeded quota, instead of hanging until the timeout.

The ReplicaSet controller reports pod-create failures on the ReplicaSet — a FailedCreate Warning event and a durable ReplicaSetReplicaFailure condition — while the deployment tracker only watched the Deployment object and its events, so these failures were never observed. StatefulSet and DaemonSet are unaffected because their controllers report FailedCreate on the object kubedog already watches.

fix: fail Deployment tracking on ReplicaSet pod-create errors

  • Read FailedCreate from the durable ReplicaSetReplicaFailure condition of the Deployment's ReplicaSets instead of transient Warning events: the condition is set while pod creation keeps failing and removed once it succeeds, so failures that predate informer startup terminate tracking immediately and no event baseline is needed. Also works with KUBEDOG_DISABLE_EVENTS=1.
  • Only ReplicaSets controlled by the tracked Deployment are considered: label selectors are not unique, so selector matching alone could pick up foreign ReplicaSets.

fix: handle ReplicaSet failure transitions consistently

  • Forced Failed payloads set IsFailed and clear IsReady; lagging FailedCreate conditions after the Deployment becomes ready are ignored.
  • Failures are deduplicated per ReplicaSet incarnation — by UID, since FailedCreate messages carry volatile quota numbers — and rearmed after condition recovery or Deployment readiness, so a failure that recurs is reported again.
  • Stale informer replays of deleted ReplicaSets are ignored (UIDs are never reused), and deleting a ReplicaSet re-evaluates the new one: the deletion may promote a template-equivalent successor that is already failing, and its failure would otherwise never be reported.
  • ReplicaSet events are delivered over unbuffered channels: additions, modifications and deletions come from a single informer goroutine but fan out into three channels, so buffering let the tracker apply an older event after a newer one and overwrite a fresh snapshot with a stale one. The informer stops sending once the consumer is gone, so a parked send cannot wedge the shared informer goroutine (regression-tested).
  • Reported failures carry a FailureMode: failures coming from events stay counted against the allowed-failures budget, while a durable ReplicaSet ReplicaFailure condition is fatal — the dynamic readiness tracker marks the resource failed, so a single pod-create error terminates tracking instead of being swallowed by the default failures allowance. Fail modes that are not supposed to fail the deploy process (IgnoreAndContinueDeployProcess) are unaffected.

Verification: go build ./..., go vet ./..., go test ./..., including new lifecycle tests for the failure transitions (dedupe, rearm, stale events, successor promotion) over fake clientsets, -race runs on the deployment/replicaset packages, and a regression test that proves the informer goroutine is released after cleanup. Originally reproduced on a kind cluster (namespace with ResourceQuota pods: 0, 1-replica Deployment): tracking hung for the full timeout before the fix and fails within seconds with the FailedCreate quota message after it.

Closes #398
Refs #216, #363

@ilya-lesikov

Copy link
Copy Markdown
Member

Thanks! But the main branch is kinda freezed right now, the development is mostly in the "1" branch. A big portion of the kubedog codebase is a mess, so I don't want to bring features to both branches

@Sharvash

Copy link
Copy Markdown
Contributor Author

Thanks! But the main branch is kinda freezed right now, the development is mostly in the "1" branch. A big portion of the kubedog codebase is a mess, so I don't want to bring features to both branches

My plan:

  • rebuild the branch on top of 1, keeping only the ReplicaSet-events commit;
  • retarget this PR to 1 (or open a fresh PR against 1, if that's easier for you to review).

Should I go ahead?

@ilya-lesikov

Copy link
Copy Markdown
Member

yes

@Sharvash
Sharvash changed the base branch from main to 1 July 24, 2026 11:30
Sharvash added a commit to Sharvash/kubedog that referenced this pull request Jul 24, 2026
Problem: during a Deployment rollout, when the child ReplicaSet cannot
create pods (for example `FailedCreate ... exceeded quota`), kubedog did
not surface the failure and tracking hung until timeout.

Root cause: the ReplicaSet controller emits these Warning events with
InvolvedObject set to the ReplicaSet, not the Deployment. The deployment
tracker only ran an event informer for the Deployment object itself, so
ReplicaSet-scoped failures were never observed. StatefulSet and DaemonSet
are unaffected because their controllers emit FailedCreate on the
controller object kubedog already watches.

The fix: start an additional event informer for each new ReplicaSet as
soon as it is discovered (tracked per RS name, so a new ReplicaSet
appearing mid-rollout gets its own informer too), feeding the same
resourceFailed channel the Deployment tracker already reacts to. Any
"Failed*" reason (including FailedCreate: exceeded quota) now interrupts
tracking, consistent with StatefulSet/DaemonSet behavior. Honors
KUBEDOG_DISABLE_EVENTS=1.

It does NOT change the event package, the failure-counting semantics of
dyntracker, or add a new immediate-abort bypass; it only widens what the
Deployment tracker observes.

Verified with go build ./... and go vet ./... on this branch. The
behavior fix was reproduced against the main branch (PR werf#399), whose
deployment tracker differs from this one only in import paths, using a
kind cluster with a ResourceQuota blocking pod creation: before,
Deployment tracking hung for the full timeout; after, it failed within
seconds with the FailedCreate reason.

Closes werf#398
Refs werf#216, werf#363

Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
@Sharvash
Sharvash force-pushed the fix/398-replicaset-quota-events branch from f5c94c5 to 405a276 Compare July 24, 2026 11:30
@Sharvash
Sharvash marked this pull request as draft July 24, 2026 13:28
Read FailedCreate from the durable ReplicaSetReplicaFailure condition instead of transient Warning events, so failures that predate informer startup terminate tracking immediately.

Only consider ReplicaSets controlled by the tracked Deployment to avoid selector collisions.

Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
Force Failed payloads to set IsFailed and clear IsReady, while ignoring lagging FailedCreate conditions after the Deployment becomes ready.

Deduplicate failures per ReplicaSet incarnation, rearm reporting after condition recovery or Deployment readiness, ignore stale events after deletion, and remove unused failure state.

Re-evaluate the new ReplicaSet once a ReplicaSet is deleted: the deletion may promote a template-equivalent successor that is already failing, and its failure would otherwise never be reported.

Deliver ReplicaSet events over unbuffered channels. Additions, modifications and deletions come from a single informer goroutine but fan out into three channels, so buffering let the tracker pick a newer modification before an older addition and overwrite a fresh snapshot with a stale one, either failing a rollout that had already recovered or losing a failure that recurred.

Qualify reported failures with a FailureMode: failures coming from events stay counted against the allowed failures count, while a durable ReplicaSet ReplicaFailure condition is reported as fatal. The dynamic readiness tracker marks the resource state of a fatal failure as failed, so a single ReplicaSet pod-create error terminates tracking instead of being swallowed by the default failures allowance.

Fail the readiness task on a failed resource state only for fail modes that are supposed to fail the deploy process, keeping IgnoreAndContinueDeployProcess unaffected.

Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
@Sharvash
Sharvash force-pushed the fix/398-replicaset-quota-events branch from 405a276 to e995853 Compare August 8, 2026 13:55
@Sharvash
Sharvash marked this pull request as ready for review August 8, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Event "exceeded quota" in replicaset should be tracked and treated as a failure

2 participants